Proactive failure handling in data processing systems

ABSTRACT

Embodiments are directed to predicting the health of a computer node using health report data and to proactively handling failures in computer network nodes. In an embodiment, a computer system monitors various health indicators for multiple nodes in a computer network. The computer system accesses stored health indicators that provide a health history for the computer network nodes. The computer system then generates a health status based on the monitored health indicators and the health history. The generated health status indicates the likelihood that the node will be healthy within a specified future time period. The computer system then leverages the generated health status to handle current or predicted failures. The computer system also presents the generated health status to a user or other entity.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims benefit of U.S. patentapplication Ser. No. 15/088,377 entitled “PROACTIVE FAILURE HANDLING INDATABASE SERVICES”, filed with the U.S. Patent and Trademark Office onApr. 1, 2016, which is a continuation of and claims benefit of U.S.patent application Ser. No. 14/537,130 entitled “PROACTIVE FAILUREHANDLING IN NETWORK NODES”, filed with the U.S. Patent and TrademarkOffice on Nov. 10, 2014, now U.S. Pat. No. 9,323,636, which is acontinuation of and claims benefit of U.S. patent application Ser. No.13/079,750 entitled “PROACTIVE FAILURE HANDLING IN DATABASE SERVICES”,filed with the U.S. Patent and Trademark Office on Apr. 4, 2011, nowU.S. Pat. No. 8,887,006, the specifications of which are incorporatedherein by reference.

BACKGROUND

Computers have become highly integrated in the workforce, in the home,in mobile devices, and many other places. Computers can process massiveamounts of information quickly and efficiently. Software applicationsdesigned to run on computer systems allow users to perform a widevariety of functions including business applications, schoolwork,entertainment and more. Software applications are often designed toperform specific tasks, such as word processor applications for draftingdocuments, or email programs for sending, receiving and organizingemail.

A database service may run in a cluster environment and may bedistributed over multiple different computer systems. A database serviceis often relied on by users and software programs alike to be availabletwenty-four hours a day, seven days a week. Accordingly, databasemanagers often implement various measures to ensure that the databaseservice is always (or nearly always) up and running. Each databasehosted in a database service has multiple replicas on different nodes sothat when failures occur on one node, a backup node is available to takeits place. Many different hardware and software failures may occur in adatabase service. One cannot anticipate the entire gamut of failuresand, hence, preparations for such failures are often inadequate.

BRIEF SUMMARY

Embodiments described herein are directed to predicting the health of acomputer node using health report data and to predicting and proactivelyhandling failures in a database service. In one embodiment, a computersystem monitors various health indicators for multiple nodes in adatabase cluster. The computer system accesses stored health indicatorsthat provide a health history for the nodes. The computer system thengenerates a health status based on the monitored health indicators andthe health history. The generated health status indicates the likelihoodthat the node will be healthy within a specified future time period. Thecomputer system also presents the predicted health status to a specifiedentity.

In another embodiment, a computer system monitors various healthindicators for multiple nodes in a database cluster. The computer systemaccesses stored health indicators that provide a health history for thedatabase cluster nodes. The computer system then generates a healthstatus based on the monitored health indicators and the health history.The generated health status indicates the likelihood that the node willbe healthy within a specified future time period. The computer systemdetermines, for at least one of the monitored nodes, that the likelihoodthat node will be healthy is below a threshold level, and transfers datastored on that node to various other nodes in the database cluster.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims, or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

To further clarify the above and other advantages and features ofembodiments of the present invention, a more particular description ofembodiments of the present invention will be rendered by reference tothe appended drawings. It is appreciated that these drawings depict onlytypical embodiments of the invention and are therefore not to beconsidered limiting of its scope. The invention will be described andexplained with additional specificity and detail through the use of theaccompanying drawings in which:

FIG. 1 illustrates a computer architecture in which embodiments of thepresent invention may operate including predicting the health of acomputer node using health report data and presenting the predictedhealth status to a specified entity.

FIG. 2 illustrates a flowchart of an example method for predicting thehealth of a computer node using health report data.

FIG. 3 illustrates a flowchart of an example method for proactivelyhandling failures in database services.

FIG. 4 illustrates an embodiment of the present invention in whichpotential failures are proactively handled.

DETAILED DESCRIPTION

Embodiments described herein are directed to predicting the health of acomputer node using health report data and to proactively handlingpredicted failures in database services. In one embodiment, a computersystem monitors various health indicators for multiple nodes in adatabase cluster. The computer system accesses stored health indicatorsthat provide a health history for the nodes in a database cluster. Thecomputer system then generates a health status based on the monitoredhealth indicators and the health history. The generated health statusindicates the likelihood that the node will be healthy within aspecified future time period. The computer system also presents thegenerated health status to a specified entity. The specified entity maybe a user, a software program, another computer system or any otherentity capable of receiving and/or viewing the health status. The usermay be any type of computer system user including an end-user, anadministrator, an information technology specialist, or any other humanparticipant that can view or interact with a computer system.

In another embodiment, a computer system monitors various healthindicators for multiple nodes in a database cluster. The computer systemaccesses stored health indicators that provide a health history for thedatabase cluster nodes. The computer system then generates a healthstatus based on the monitored health indicators and the health history.The generated health status indicates the likelihood that the node willbe healthy within a specified future time period. The computer systemdetermines, for at least one of the monitored nodes, that the likelihoodthat node will be healthy is below a threshold level, and transfersportions of data stored on the monitored node to various other nodes inthe database cluster. It also prevents new data from being placed on thenode which is predicted to fail.

The following discussion now refers to a number of methods and methodacts that may be performed. It should be noted, that although the methodacts may be discussed in a certain order or illustrated in a flow chartas occurring in a particular order, no particular ordering isnecessarily required unless specifically stated, or required because anact is dependent on another act being completed prior to the act beingperformed.

Embodiments of the present invention may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentinvention also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arecomputer storage media. Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, embodiments of the invention can compriseat least two distinctly different kinds of computer-readable media:computer storage media and transmission media.

Computer storage media includes RAM, ROM, EEPROM, CD-ROM, solid statedevices (SSDs) or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium which can be used tostore desired program code means in the form of computer-executableinstructions or data structures and which can be accessed by a generalpurpose or special purpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry data or desired program code means in theform of computer-executable instructions or data structures and whichcan be accessed by a general purpose or special purpose computer.Combinations of the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to computerstorage media (or vice versa). For example, computer-executableinstructions or data structures received over a network or data link canbe buffered in RAM within a network interface module (e.g., a “NIC”),and then eventually transferred to computer system RAM and/or to lessvolatile computer storage media at a computer system. Thus, it should beunderstood that computer storage media can be included in computersystem components that also (or even primarily) utilize transmissionmedia.

Computer-executable instructions comprise, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. The computer executable instructions may be, forexample, binaries, intermediate format instructions such as assemblylanguage, or even source code. Although the subject matter has beendescribed in language specific to structural features and/ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thedescribed features or acts described above. Rather, the describedfeatures and acts are disclosed as example forms of implementing theclaims.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The invention may also bepracticed in distributed system environments where local and remotecomputer systems, which are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network, both perform tasks (e.g. cloud computing,cloud services and the like). In a distributed system environment,program modules may be located in both local and remote memory storagedevices.

FIG. 1 illustrates a computer architecture 100 in which the principlesof the present invention may be employed. Computer architecture 100includes a database cluster with multiple database nodes 110A-110D. Thedatabase cluster may be the backend of a distributed database service.Each database node may be configured to perform database functions andmay operate in conjunction with the other nodes of the database cluster.While four nodes are shown in FIG. 1, it will be understood thatsubstantially any number of nodes may be used in the database cluster.

Each node may include a health monitoring module 115 that performshealth monitoring functions. For instance, the health monitoring modulemay monitor the current networking conditions, software failures,hardware failures, error messages indicating errors in software and/orhardware components, or any other indications of a computer node'shealth. Each node may report their current health status 111 to node110A. Node 110A maintains a health history 116 of each node, by storingsnapshots of their health status 111 in a persisted fashion. In somecases, the health monitoring module may be implemented as a part of thedistributed service. In other cases, a cluster management service may beused with the cluster which automates provisioning, monitoring, andrepairing tasks.

Database node 110A can use the other nodes' health statuses to predictor otherwise determine the nodes' future health. For example, if node110B has been experiencing hardware or software failures (currentfailures or past failures as stored in health history 116), node 110Amay be able to predict whether node will be healthy or unhealthy in thefuture. As used herein, a “failure” may refer to any transition from ahealthy state to an unhealthy state. This takes into account both priornode state (e.g. stored health history 116) and current health statusinformation collected by the cluster management service.

In some embodiments, the number of failures may be counted for each nodein a recent, specified timeframe. If the number of failures exceeds athreshold amount, the node is predicted to be unstable or otherwiseunhealthy for the near future. In some cases, an error level may beassigned to each failure that occurs, which encodes severity and natureof the failure. If a node is showing failures with an error level abovea specified threshold, (for example, if the node is showing hardwarefailures), the node is also predicted to be unhealthy for the nearfuture, without requiring the failure count to exceed the aforementionedthreshold. If neither of the above applies to a node, it is predicted tobe healthy. The nodes' health status may be continually re-evaluated, tooptimize the accuracy of the node's predicted health status.

Node 110A may present the predicted health status 126 of each node to aspecified entity 105. The predicted health status 126 may be leveragedby entity 105 in several ways. For example, the predicted health statusmay be used in forming a better view of the health status of the entirecluster. It may be used to distinguish between persistent and transientnode failures. It may also be used to proactively prepare for nodefailures that may happen in the future. Further details will be enclosedin the descriptions of other embodiments of this invention wherepredicted node failures are proactively and automatically handled.

In view of the systems and architectures described above, methodologiesthat may be implemented in accordance with the disclosed subject matterwill be better appreciated with reference to the flow charts of FIGS. 2and 3. For purposes of simplicity of explanation, the methodologies areshown and described as a series of blocks. However, it should beunderstood and appreciated that the claimed subject matter is notlimited by the order of the blocks, as some blocks may occur indifferent orders and/or concurrently with other blocks from what isdepicted and described herein. Moreover, not all illustrated blocks maybe required to implement the methodologies described hereinafter.

FIG. 2 illustrates a flowchart of a method 200 for predicting the healthof a computer node using health report data. The method 200 will now bedescribed with frequent reference to the components and data ofenvironment 100.

Method 200 includes an act of monitoring one or more health indicatorsfor a plurality of nodes in a database cluster (act 210). For example,health monitoring module 115 may monitor various health indicators fordatabase nodes 110A-110D. The health monitoring may include identifyingand logging the occurrence of hardware errors, software errors,processing load or other indicators of a computer node's health.

Method 200 includes an act of accessing one or more stored healthindicators that provide a health history for the database nodes (act220). For example, the current health status sent by the database nodesmay be stored over a period of time in health history 116. These storedhealth snapshots may represent a node's health over a period of time.For instance, the stored health history may show which hardware andsoftware errors have occurred over a specified period of time. Thehealth history may further show patterns such as an increasing ordecreasing number of errors, which may indicate that the node is headingtoward failure, or is showing signs of durability.

Method 200 includes an act of generating a predicted health status basedon the monitored health indicators and the health history, wherein thepredicted health status indicates the likelihood that the node will behealthy within a specified future time period (act 230). For example,the health prediction module may generate a health status report 126based on the monitored health indicators 111 and on the stored healthhistory. The health status report indicates the likelihood, for eachmonitored node, that the node will be healthy in the future. The reportmay thus be a prediction of how well the node will perform in thefuture. The prediction may be based on errors that have occurred in thepast, or that are currently occurring. The prediction may be based onpatterns of failures, severity of failures, types of failures, frequencyof occurrence, average processing load, or other factors.

Method 200 includes an act of presenting the generated health status toa specified entity (act 240). For example, the health status 126predicted by health prediction module 120 may be presented to entity 105by presentation module 125. The presentation may illustrate which errorshave occurred, when the errors occurred, what type of errors occurred,and so forth. The presentation may further show, for each node in thecluster, what the likelihood is that the node will keep functioningcorrectly in the near future. The user may then use the information inthe presentation to proactively manage the nodes in the databasecluster.

In some embodiments, the database cluster may proactively handlepredicted node failures, in order to maintain high availability of theservice, at least in some cases without much human intervention. In suchembodiments, a blacklist may be devised. Members of the blacklist arenodes that may result in undesirable outcome if they continue or startto participate in functionality of the cluster. The health predictionmechanism described above may be used as a basis to maintain themembership of the blacklist. For a node that is not currently on theblacklist, if the predicted health 126 indicates likely future failures,it is put onto the blacklist. Once a node joins the blacklist, aprobationary period of some specified time may be implemented before itcan be removed from the blacklist. Throughout the probationary period,the current health status 111 and the predicted health status 126 of thenode both have to satisfy some specified criteria. If either healthstatus fails to satisfy the criteria, the probationary period may bereset, that is, the node has to remain on the blacklist for at leastanother probationary period.

Distinguishing between persistent and transient node failures may beaided by the use of the blacklist. When a node failure happens, if thenode is not on the blacklist, it may indicate that the failure istransient and the node may become healthy again within an acceptabletimeframe. If the failed node is on the blacklist, it may indicate thatthe failure is persistent and the node may not recover anytime soon. Indatabase services, preparing one or more backup nodes to fully replace afailed node may be an expensive operation. It may involve, among otherthings, transferring all the data that used to be hosted by the failednode onto the backup node(s). Thus it may be desirable to avoid thisoperation as much as possible. For transient node failures, the databasecluster may temporarily work around the failed node without fullyevicting it. Then when the failed node becomes healthy again, it canrejoin the cluster. For persistent node failures, one or more backupnodes may have to be called upon to take the place of the failed node tomaintain database redundancy.

Thus, the type of node failure may be used to determine which actionsare to be taken in response. When transient failures occur, userrequests are temporarily redirected to other nodes hosting the same dataportions as the failed node, and a grace period may be granted for thenode to recover. Only if the node does not recover within the graceperiod, one or more backup nodes may be called upon to replace it. Whenpersistent failures occur, it is pointless to wait for the node torecover, so one or more backup nodes may be called upon immediately.

Blacklist membership itself can also be a precursor or indicator offuture failures. For instance, a node may appear usable at the moment itis being blacklisted. This may occur due to environmental issues, suchas hard drive bad sectors. Such errors may take a while before affectingsoftware running on that node. However, by proactively transferring dataaway from the node, the database management service may reduce theimpact of future errors. Moreover, by not transferring new data to thenode, the failure may be contained. These concepts will be illustratedbelow with regard to FIGS. 3 and 4.

FIG. 3 illustrates a flowchart of a method 300 for proactively handlingfailures in database services. The method 300 will now be described withfrequent reference to the components and data of environment 400.

Method 300 includes an act of monitoring one or more health indicatorsfor a plurality of nodes in a database cluster (act 310). For example,health monitoring module 415 may monitor health indicators for any oneor more of database nodes 410A, 410B, 410C and 410D. As explained above,the health monitoring may include identifying and logging the occurrenceof hardware errors, software errors, processing load or any otherindicators of a computer node's health. The current health status 411reported by the monitored nodes may be stored in health history 416. Thecurrent health status and health history may be accessed by the healthprediction module 420 (act 320). Using the current health status (i.e.the monitored health indicators) and the stored health history, module420 may generate a health status that indicates the likelihood that themonitored nodes will be healthy within a specified future time period(act 330).

For example, database node 410D may have experienced multiple errorsover a period of time. These errors are monitored and logged. If theerrors are serious enough, or if the frequency of occurrence is highenough, the node will be predicted to be unhealthy in the near future.

Method 300 also includes an act of determining, for at least one of themonitored nodes, that the likelihood that node will be healthy is belowa threshold level (act 340). Based on the determination, the node may beblacklisted, and a blacklist manager may update the membership of theblacklist. For example, blacklist management module 430 may determinefor database node 410D that the node has a low likelihood of beinghealthy in the future. This determination may be based on the currenthealth status, or on error patterns determined from the stored healthhistory 416, or from both.

Method 300 includes an act of transferring one or more portions of datastored on the monitored node to one or more other nodes in the databasecluster (act 350) and an act of preventing the monitored node fromstoring new data (act 360). For example, based on the determination madeby the blacklist management module 430, the data on node 410D may betransferred to any of nodes 410A, 410B and 410C. Node 410D may beblacklisted and prevented from storing new data.

Node 410D may appear either usable or unusable when it is blacklisted.The states may be referred to as “Up and Blacklisted” and “Down andBlacklisted” respectively. In cases where the node is down andblacklisted, one or more backup nodes may be called upon immediately anddata transfer module 435 may indicate to other nodes in the clusterhosting the same data portions as node 410D, that the data stored onnode 410D is to be transferred to the backup node(s). In cases where thenode is up and blacklisted, it may continue to provide access to thedata already stored on it, while its data is being transferred to thebackup node(s) (e.g. as part of a background process). However, no newdata is allowed to be stored on the node.

The current blacklist, as well as the current health status for eachnode may be provided to the user 405 in health status report 426. Usingthis report, the user may make efficient database management decisions,and may proactively handle errors on the various database nodes as asupplement to the automated responses.

Accordingly, methods, systems and computer program products are providedwhich predict the health of a computer node using health report data.Moreover, methods, systems and computer program products are providedwhich proactively handle failures in database services.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

We claim:
 1. A computer system for predicting the health of a pluralityof nodes by using health report data, the computer system comprising oneor more processors executing computer executable instructions whichcause the computer system to perform the following: monitors one or morehealth indicators for a plurality of nodes; accesses one or more storedhealth indicators that provide a health history for one or more of themonitored plurality of nodes; based on both the monitored healthindicators and the stored health history, predicts a health status,wherein the predicted health status indicates for at least one of themonitored plurality of nodes that the at least one monitored node willbe healthy or unhealthy in the future; and presents the predicted healthstatus to a specified entity.
 2. The computer system of claim 1, whereinthe monitored nodes are nodes of a data processing system, and whereinthe predicted health status comprises a determination that there is aprobability that the at least one monitored node will be healthy, andwherein the determination is below a threshold level; and transfers oneor more portions of data stored on the at least one monitored node toone or more other nodes of the data processing system.
 3. The computersystem of claim 2, wherein the computer system further performs thefollowing: prevents the at least one monitored node from storing newdata.
 4. The computer system of claim 2, wherein making thedetermination that the probability that the at least one monitored nodewill be healthy, and wherein the determination is below a thresholdlevel, comprises determining that the at least one monitored node is ina critical state.
 5. The computer system of claim 2, wherein making thedetermination that the probability that the at least one monitored nodewill be healthy, and wherein the determination is below a thresholdlevel, comprises determining that the at least one monitored node hasexperienced one or more failures within a specified time period.
 6. Thecomputer system of claim 5, wherein the computer system further performsthe following: assigns an error level to each of the failures.
 7. Thecomputer system of claim 6, wherein the computer system further performsthe following: determines that a threshold number of the failures arebeyond a specified error level, such that the at least one monitorednode is blacklisted.
 8. The computer system of claim 7, wherein anymonitored node that has a threshold number of failures beyond aspecified error level are blacklisted, regardless of a monitored node'shealth history.
 9. The computer system of claim 7, wherein blacklistednodes are put on probation for a specified amount of time to determinewhether errors occur during probation.
 10. The computer system of claim9, wherein the computer system further performs the following: upondetermining for a monitored node that was blacklisted that theprobationary period is complete and that no further errors haveoccurred, allowing the monitored node that was blacklisted to continuestoring new data and removing the monitored node from the blacklist. 11.The computer system of claim 9, wherein the computer system furtherperforms the following: upon determining for a monitored node that wasblacklisted that the probationary period is complete and that one ormore further errors have occurred, preventing the monitored node thatwas blacklisted from storing new data.
 12. The computer system of claim11, wherein the computer system further performs the following:relocates data portions that are hosted on the monitored node that isprevented from storing new data.
 13. The computer system of claim 1,wherein the predicted health status indicates for at least one of themonitored plurality of nodes that there is a probability that the atleast one monitored node will be healthy in the future.
 14. The computersystem of claim 1, wherein the predicted health status indicates for atleast one of the monitored plurality of nodes that there is aprobability that at least one monitored node will be healthy within aspecified future time period.
 15. The computer system of claim 1,wherein the predicted health status indicates for at least one of themonitored plurality of nodes that there is a probability that at leastone monitored node will be unhealthy in the future.
 16. Acomputer-implemented method for proactively handling failures in adistributed processing system comprising a plurality of nodes, thecomputer-implemented method being performed by one or more processorsexecuting computer executable instructions for the computer-implementedmethod, and the computer-implemented method comprising: monitoring oneor more health indicators for a plurality of nodes of a distributedprocessing system; accessing one or more stored health indicators thatprovide a health history for the one or more of the monitored nodes ofthe distributed processing system; predicting a health status based onthe monitored health indicators and the health history, wherein thepredicted health status indicates that at least one of the one or moremonitored nodes of the distributed processing system will be healthy orunhealthy in the future; determining, for at least one of the monitorednodes of the distributed processing system, that a threshold number offailures have occurred that are beyond a specified error level; based onthe determination, blacklisting the at least one monitored node forwhich the determination was made; transferring one or more portions ofdata stored from the at least one monitored node that is blacklisted toone or more of other nodes of the distributed processing system; andpreventing the node that was blacklisted from storing new data.
 17. Thecomputer-implemented method of claim 16, wherein the node that isblacklisted is categorized as up and blacklisted, such that theblacklisted node remains used for storing data while the data istransferred to other nodes of the distributed processing system, and nonew data is stored on the node that is up and blacklisted.
 18. Thecomputer-implemented method of claim 16, wherein the node that isblacklisted is categorized as down and blacklisted, such that theblacklisted node is no longer used for storing data, and data istransferred from the blacklisted node to other nodes of the distributedprocessing, and no new data is stored on the node that is down andblacklisted.
 19. The computer-implemented method of claim 18, whereinthe data is transferred without waiting for a probationary period. 20.The computer-implemented method of claim 16, wherein the predictedhealth status indicates for at least one of the monitored plurality ofnodes that there is a probability that the at least one monitored nodewill be healthy in the future.
 21. The computer-implemented method ofclaim 16, wherein the predicted health status indicates for at least oneof the monitored plurality of nodes that there is a probability that theat least one monitored node will be healthy within a specified futuretime period.
 22. The computer-implemented method of claim 16, whereinthe predicted health status indicates for at least one of the monitoredplurality of nodes that there is a probability that the at least onemonitored node will be unhealthy in the future.
 23. A system comprising:a processing system comprised of a plurality of nodes; one or morecomputer-readable storage hardware media, excluding transmission media,having stored thereon computer-executable instructions that, whenexecuted by one or more processors, cause the system to be configuredwith an architecture that proactively handles failures in the pluralityof nodes by using health indicators, and wherein the architecture isconfigured to perform the following: monitor one or more healthindicators for a plurality of nodes of a processing system; access oneor more stored health indicators that provide a health history for oneor more of the monitored plurality of nodes; based on both the monitoredhealth indicators and the stored health history, predict a healthstatus, wherein the predicted health status indicates for at least oneof the monitored plurality of nodes that the at least one monitored nodewill be healthy or unhealthy in the future; and present the predictedhealth status to a specified entity.
 24. The system of claim 23, whereinthe architecture is further configured to blacklist at least one of themonitored nodes upon determining that a threshold number of failureshave occurred that are beyond a specified error level.
 25. The system ofclaim 24, wherein the architecture is further configured to transfer oneor more portions of data stored on the monitored node that isblacklisted to one or more other nodes of the processing system.
 26. Thesystem of claim 25, wherein the node that is blacklisted is categorizedas up and blacklisted, such that the categorized node remains used forstoring data while the data is transferred to other nodes of theprocessing systems, and no new data is stored on the node that is up andblacklisted.
 27. The system of claim 23, wherein the predicted healthstatus indicates for at least one of the monitored plurality of nodesthat there is a probability that the at least one monitored node will behealthy in the future.
 28. The system of claim 23, wherein the predictedhealth status indicates for at least one of the monitored plurality ofnodes that there is a probability that the at least one monitored nodewill be healthy within a specified future time period.
 29. The system ofclaim 23, wherein the predicted health status indicates for at least oneof the monitored plurality of nodes that there is a probability that theat least one monitored node will be unhealthy in the future.